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ABSTRACT 

This paper reports the results of two semesters of 
experience using computer-assisted instruction (CAI) to teach topics 
in program evaluation to undergraduate and graduate psychology 
students at California State University, Long Beach. (The topics 
addressed are models of evaluation, evaluability assessment, needs 
assessment, experimental and quasi-experimental design, management 
information systems, roles and conflicts, and cost-based evaluation 
methods.) A description of the structure of the program evaluation 
course in which the software is used is followed by a discussion of 
the graphical engine, a program compiled in Turbo Pascal which drives 
the tutorials and is capable of presenting graphics and text, taking 
student responses, and providing feedback. Results are reported of 
three preliminary evaluations involving 22 seniors and first year 
graduate students. It is noted that the CAI tutorials have not yet 
shown evidence of superiority over a more conventional textual 
presentation. (GL) 
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OERI position or policy Thls paper reports the results of two semesters of 

experience using computer aided instruction (CAI) to teach 
.(JsJ program evaluation to undergraduate and graduate psychology 

students. The purposes of the paper are to describe the 
^ CAI package, report initial results and to solicit col- 

leagues who may wish to beta test the modules at no cost 
^ and/or contribute to the development of the modules. 
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structure of the Program Evaluation Course 



The first author has been teaching program evalua- 
tion at both graduate and undergraduate university levels 
ptj since 1978. The present structure of the program evalua- 

tion course involves two hours of lecture and three hours 
of laboratory each week. In the early years of teaching 
the course, the first author developed a series of paper 
and pencil simulations of program evaluation problems which 
became the focus of the laboratories. In addition, stu- 
dents used laboratory time to work on and ask questions 
about semester projects involving evaluation design and 
practice . 

Beginning in the fall of 1988, the first author be- 
gan to use computer aided instructior in laboratory portion 
of the course. This was made possible by the implementa- 
tion of a microcomputer laboratory in the psychology de- 
partment equipped with 12 Zenith 80286 class computers. 
Prior to this semester, the first author has undertaken to 
complete the creation of two separate products, a computer 
program which would drive a graphical, interactive learning 
environment for students, and a set of tutorials which 
would cover the basic concepts of program evaluation. The 
first of these tasks involvec converting paper and pencil 
simulations to a format compatible with the CAI environment 
and the second involved programming in Turbo Pascal, Ver- 
sion 3.0. This was later upgraded to Turbo Pascal, Version 
5.0. 



^ ^ The first author is resporsible for the development of 

the CAI tutorials described in this paper including the 

^ programming and the content of the tutorials. The second 

author contributed significantly to the evaluation of the 
tutorials. 
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The Graphical Engine 



The graphical engine is a program compiled in Turbo 
Pascal which drives the tutorials. This program is capable 
of presenting to students graphics and text, taking student 
responses, and providing feedback. The graphical capabili- 
ties of the program include bargraphs , piecharts , experi - 
mental plots, crosstabulations , point plots, line graphs, 
and plotting of algebraic equations. The graphics appear 
in one or two windows on screen while text occupies a third 
window. The program is capable of modifying graphics based 
upon student input , providing students with "what- if*' expe- 
riences in modifying parameters. For example, to illus- 
trate power analysis in survey sampl ing , students could en- 
ter sample sizes a resultant plot of confidence levels and 
intervals . 

The graphics are each called with one to three sim- 
ple statements embedded iiito the text of the tutorials (see 
below) . The principle advantage of this compiled 
graphics-based "engine" for the tutorials is the ease with 
which tutorials can be constructed . A tutorial can be 
written in as little as ten minutes including any of the 
graphics noted. This is designed to overcome two major 
hurdles to CAI de-'elopment : the time required to write good 
tutorials and the difficulty of including high resolution 
graphics. Other advantages associated with the package no 
royalties on the compiled version, support for a variety of 
graphical devices including CGA, EGA, VGA and Hercules, 
compatibility with PC, XT, AT, and 386 class machines. The 
source code could be ported to another computing environ- 
ment such as the Macintosh because Pascal is a high level 
language although this has not yet been undertaken. 

The Tutorial Co/itents 

The second component of the CAI modules is a set of 
tutorials covering the following topics in program evalua- 
tion: 

-models of evaluation 
-evaluability assessment 
-needs assessment 

-experimental and quasi-experimental des'^gn 
-management information systems 
-roles and conflicts 
-cost-based evaluation methods 



These tutorials are made available to the graphical 
engine in the form of ASCII text files. Graphical com- 
mands, typically one line of instruction to the grapliical 
engine, are embedded in the tutorials and are not displayed 
but are interpreted by the engine. Because any word proc- 
essor can produce and edit ASCII text and because the 
graphics are easily embedded, an instructor can use the ex- 



isting tutorials, modify them to suit, or create new ones 
quickly. 

The tutorials were developed in two general for- 
mats. One is a multiple choice format illustrated with 
graphics which provides students information, asks the stu- 
dents to make choices , and provides immediate feedback. 
The second format is to provide students with essay task 
focused on the material offered in the first format and 
providing students with immediate feedback in the form of a 
sample essay response. The second format could be accom- 
plished with any word processor. Although the tutorials 
were originally conceived as being only in the first for- 
mat , the limitations of multiple choice input prompted the 
first author to develop both types of experiences and to 
link them in the laboratories. 

These tutorials are designed to be compatible with 
the textbook Program Evaluation. Methods and Case Studies 
by Posavac and Carey but they are general enough to be used 
with any evaluation text book. The tutorials have been 
adapted from similar tutorials used by the author for years 
in pencil and paper environment. 



Methods 



Sub.lects 

Subjects were twenty-two seniors and first year 
graduate students enrolled in a program evaluation course. 
Nineteen were female and three were male. All subjects 
were psychology students at California State University, 
Long Beach. 

Procedures 

Three evaluation studies took place during the sec- 
ci.^d to fourth week of the semester during hours assigned to 
laboratories for the course. Three different evaluation 
activities were carried out. 

In the first study, students participated in a com-- 
puter version or a pencil and paper version of the same lab 
exercise, focused upon measurement strategies in evaluation 
research. Students were divided into two groups on the ba- 
sis of the last digit of social securxcy numbers and each 
group was comprised of 10 S*s. Students in one group re- 
ceived the computer tutorials while students in the com- 
parison group received the same material in a pencil and 
paper format. Students completed a 20-item quiz over the 
measurement and also wrote an essay answer to an open-ended 
problem over the ipaterial. The essay responses were inde- 
pendently judged by two raters according to the 
presence/absence of 13 criteria deemed important. 

For the second study, parallel forms of the same 
quiz regarding needs assessment were used to conduct a 
pretest-posttest evaluation. All students received the 
computer tutorials. Students were divided into two groups 
based upon social security number with one group receiving 
a 17 -item multiple choice pretest while the other received 
only a 17-item posttest. Parallel forms were used with 
students in a Tuesday lab receiving Form A as a pretest 
with Form B as the posttest and those in the Thursday lab 
receiving the reverse. The 17 item multiple choice quizzes 
were scored for correct answers. One item on one form was 
dropped because of ambiguity in wording. Scores were ad-- 
justed for the number items. 

For the third study, students were surveys at a 
point after which all had received both computer and pen-- 
cil/paper laboratory exercises ' for their perceptions. 

Results 

Study 1^ Interrater reliability for the scoring of 
the essay responses was satisfactory (r=0.81, p<.001). For 
each condition, the correct responses on the thirteen 
criteria were recorded and tested for significance. The 
difference was not significant |t=0.14, df=18, p<.5, one- 



tailed). On the multiple choice quiz, the difference which 
favored students who did not take the pretest, was also not 
significant (t=0.18, df=18, p<0.425, one-tailed). 

Study 2* There wes no significant difference as a 
function of receiving the pretest (t=~1.78, df=18, p=0.092, 
two-tailed). There was also not a significant difference 
from pretest to posttest for students who received the 
tutorials (t=-1.27, df=10, p=0.117, one-tailed). However, 
this difference was in the expected direction and the mag- 
nitude of the difference was 36 percent of the pretest 
standard deviation, or an effect size of about 0.36. With 
the small sample, this was not significant. 

Study 3. Students who had participated in both 
types of labs rated both on the following dimensions: 
clarity, helpfulness , fun, amount learned , control , ease of 
use, clarity of instructions, and overall rating. Only two 
differences were significant in t-test comparisons 
(two-tailed). Computer labs were rated as less controlla- 
ble and as more helpful. In open-ended comments which were 
"oded for clearly stated preferences for one lab over the 
other, five preferred the computers, three the pencil and 
paper and, thirteen did not express a clear preference. 



Discussion 



The results of the preliminary evaluation are mixed 
at best. There is no evidence that students learn more 
from the graphical environment compared to simple textual 
presentation. The pretest-posttest measure clxd trend in 
the expected direction; in fact the gain from pre to post 
was about 36 percent of the pretest standard deviation (a 
an even larger fraction of the posttest standard 
deviation). Students do rate the computer labs as signifi- 
cantly more helpful but the absence of significance for 
such dimensions as graphical presentation is puzzling. 
Students rated the computer labs as significantly less 
positive on the control dimension, which may be a key di- 
mension which prevents the computer labs from being more 
effective . 

A number of variables may have influenced the re- 
sults which provide important information about the use of 
computer tutorials. First, the comparison group test is a 
rather stringent test of the CAI modules since student re- 
ceived the same material not only in a pencil and paper 
format but in the textbook and lecture. Nevertheless, to 
justify use, CAI should eventually show this form of supe- 
riority. Second, several students in the course expressed 
some aegree of computer phobia ranging to extreme dislike 
for computers. It is likely that ratings are influenced by 
this anxiety about the use of computers despite the fact 
that the tutorials were very simple? to use. Third, there 
were a few inevitable bugs in the tutorials which occasion- 
ally caused frustration when the text written by a student 
was lost by a computer, for example. Fourth, and perhaps 
very importantly, the computers v;ere perceived as less con- 
trollable than the alternative lab. This was partly a 
function of the text display procedures which did not allow 
students to easily move backwards within tutorials. This 
perceived lack of controllability may have interacted with 
computer anxiety for some students to reduce the effective- 
ness of the CAI approach. The newer version of the Graphi- 
cal Educational Environment takes a different and more 
flexible approach to student control of the display. Con- 
trollability is such an important dimension of human per- 
formance that maximizing this dimension probably should be 
a paramount goal of CAI programs. It may well be so impor- 
tant and interact so strongly with student learning styles 
that CAI will never be effective for some students. 

One non-rigorous source of data is the impressions 
of the instructor in interacting with students about the 
tutorials. The tutorials are experienced as relatively 
easy to use by the students. There are relatively few in- 
stances of students getting **stuck** and not knowing how to 
proceed* The tutorials provoke good questions from the 
students, although this is a function of the content more 
than the ccuipi^ter display. The instructor has often been 



pleasantly impressed at a question which identifies an am- 
biguity or comrlexity of the tutorials. Most of the ques- 
tions received from students are of this nature, indicating 
that the tutorials generate a fairly high degree of compre- 
hension and involvement from students. 

On the other hand, the CAI tutorials are by their 
nature , relatively asocial experiences . This was espec i al- 
ly an interesting contrast for the first author who used to 
teach the laboratories as a discussion group experience fo- 
cused upon simulations and for the second author who had 
taken the course in this format. The computer labs' atmos- 
phere of absorbed individual concentration is a striking 
contrast to the previous format * s atmosphere of 1 ively in- 
terested discussion . 

However, one weakness of the previous format which 
the first author had perceived and which was an impetus for 
the CAI approach was uneven understanding of differing stu- 
dents. In a group format, there is a limit upon how much 
repetition can occur for students who are having difficulty 
with comprehension. There is not similar limit in the CAI 
approach since students can review as much as desired. 

In sum, the CAI tutorials have not yet shown evi- 
dence of superiority to a more conventional textual presen- 
tation but this evaluation has focusei' efforts to improve 
the program upon maximising controllability and flexibility 
for students. It is possible that a next generation of the 
program will be a useful adjunct to teaching -orogram evalu- 
ation. 
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